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BACKGROUND OF THE INVENTION 

1. CROSS-REFERENCE TO RELATED APPLICATIONS) 

5 The present application claims the benefit of priority from pending U.S. 

Provisional Patent Application No. 60/472,626, entitled "Physical/Digital Input Devices 
For 3D Spatial Manipulation M , filed on May 22, 2003, which is herein incorporated by 
reference in its entirety. 

10 2. FIELD OF THE INVENTION 

The present invention relates to the field of using both physical and digital input 
methodologies to implement both two and three-dimensional spatial manipulation and 
two and three-dimensional entertainment. 

15 

Portions of the disclosure of this patent document contain material that is subject 
to copyright protection. The copyright owner has no objection to the facsimile 
reproduction by anyone of the patent document or the patent disclosure as it appears in 
the Patent and Trademark Office file or records, but otherwise reserves all rights 
20 whatsoever. 

3. BACKGROUND ART 

Computers have become a resource for modeling complex spatial structural 
25 designs. With the progression in technology, very complex three-dimensional designs 
have become ubiquitous in many fields. These three-dimensional depictions are used to 
represent designs of varied scale, ranging from nanoscale molecular machines to large 
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architectural city structures. For advanced progress of the creation of these designs, it is 
very important for a designer to be able to translate a three-dimensional design idea into a 
digital representation. 

5 Currently, many designers use two-dimensional graphical user interfaces (GUIs) 

to model their three-dimensional designs. A two-dimensional GUI consists of some type 
of cursor controller, for example a mouse, which usually is only able to control two 
coordinates, namely X and Y. Since a typical mouse can only be moved in a two- 
dimensional plane, the GUI has only two input degrees of freedom (DOF). Thus, the 
10 position of the mouse can only be translated along the X axis and along the Y axis. 

A disadvantage to these two-dimensional GUIs is that they rarely allow the 
designer to seamlessly create a digital representation of a three-dimensional design. 
There are a couple of reasons why two-dimensional GUIs present interruptions to the 
15 design process. One reason is that since two-dimensional GUIs only translate position 
along the X axis and Y axis, a designer has to stop the digital design process and rotate 
the entire virtual scene to draw virtual objects along the Z axis. These interruptions can 
have detrimental effects to the design process by breaking the designer's stream-of- 
thoughts during the design process by making the design process longer. 

20 

The other reason is based on the fact that most of these two-dimensional GUIs use 
one generic mouse as an input device. A generic mouse usually contains only two 
buttons, which are available to be used for modal selection. This presents another 
obstacle to the design process because since design software usually contains many 
25 different drawing tools and there are only two available mouse buttons, the selection of a 
drawing tool can usually only be achieved by some sort of awkward means, e.g., by 
simultaneously depressing the Alt-Shift keyboard buttons along with one of the mouse 
0007975-0048/CIT-3892 3 

LOSANGELES 105062v4 



buttons. Since this awkward activation requires additional thought, it can also cause the 
deleterious effects of breaking the designer's thought sequence and exhausting the 
designer. 

5 Thus, since the commonly used two-dimensional GUIs present these obstacles in 

three-dimensional design, there is a need for a system that allows for easier manipulation 
and construction of three-dimensional digital designs and for easier modal selections. 

These devices, which allow for easier spatial manipulation, can also be applied to 
10 entertainment. In particular, video games in arcades, on personal computers, and on 

consoles, to name a few locations, often involve the rapid and complex manipulation of 
spatial objects such as, and not limited to, characters (also known as avatars), weapons, 
projectiles, prizes, tools, puzzle pieces, enemies, and other objects that have value in a 
gaming scenario. These same devices can be used to create a new type of entertainment 
15 experience which extends and enhances gaming. 

Thus, there is a need for a system that not only allows for easier manipulation and 
construction of both two and three-dimensional digital design, but also for easier 
manipulation and construction of both two and three-dimensional entertainment, for 
20 example, gaming. 
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4 



SUMMARY OF THE INVENTION 



The present invention provides improved interface tools, both physical and digital 
for performing two and three-dimensional spatial manipulation and two and three- 
5 dimensional gaming. One embodiment of the invention is a set of three-dimensional 
multifunction tangible tools. These tools are shaped to specific forms. The forms mimic 
the tools' primary function, so the user gains an immediate intuition as to how each tool 
operates. For example, one of the tools is shaped to resemble a pair of kitchen tongs, 
which gives the user the immediate perception that they are to be used for grabbing. In 
10 addition, these tangible tools also have the advantage that they are able to be remapped to 
fulfill different functions. 



Another embodiment of the invention is a methodology for using two physical or 
digital input devices in conjunction to alter two and three-dimensional virtual objects. 
15 For example, two grabbing tangible tools can be used together to alter the structure of one 
virtual object. 



An additional embodiment of the invention is a methodology for associating 
physical components to two or three-dimensional virtual components. Note that a virtual 

20 component is a software representation of a physical component, and the software 

representation has a two or three-dimensional rendering associated with it. For example, 
a virtual component is a cursor on a computer screen. This methodology involves two 
concepts. A first concept involves the notion of being able to alter the relationship 
between a physical input methodology and its corresponding three-dimensional virtual 

25 components. A second concept involves various ways to map physical components to 
virtual components. According to another embodiment, the physical components are able 
to be mapped to one or more virtual components. According to another embodiment, this 
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concept allows changing the type of two or three-dimensional virtual component(s) that is 
mapped to the physical component. For example, a user can change a physical 
component's virtual component representation from a spherical ball to a cube. 



5 According to another embodiment, the invention presents several novel tools to 

alter and generate three-dimensional objects. Examples of these tools include a three 
dimensional eraser tool, which removes a region of a surface; a deformation tool, which 
is used to deform the geometry of a virtual object; a spray-painting tool, which is used to 
"spray" a three-dimensional virtual object with virtual paint; a smoothing tool, which is 
10 used to smooth the surface of a virtual object; and a texture creation tool, which is used to 
"spray" texture onto a three-dimensional virtual object. 

According to another embodiment, the invention presents several novel means of 
interacting with video games using the tools mentioned above as a replacement for the 
15 standard joystick. For example, using the tool to grab things in space, to shoot, cut, 

place, rotate, and twist objects, to specify regions of space for the purposes of gameplay, 
drawing strokes in three-dimensional space as a component of gameplay, and using these 
tools for navigation of game-space in a variety of forms. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



Figure 1 illustrates three degrees of freedom of a generic rigid body in space. 

5 Figure 2 illustrates six degrees of freedom of a generic rigid body in space. 

Figure 3 is an illustration depicting a grabbing tangible tool, according to one 
embodiment of the present invention. 

10 Figure 4 is a flow chart showing the steps for checking for the position of a virtual 

component. 

Figure 5 is a flow chart showing the steps involved with activation of a virtual 

menu. 

15 

Figure 6 is an illustration depicting a pointing tangible tool, according to one 
embodiment of the present invention. 

Figure 7 is an illustration depicting a gripping tangible tool, according to one 
20 embodiment of the present invention. 

Figure 8 is an illustration depicting the virtual menu of the gripping tangible tool, 
according to one embodiment of the present invention. 

25 Figure 9 is an illustration of the user moving the tool towards one icon of the 

virtual menu of the gripping tangible tool, according to one embodiment of the present 
invention. 
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Figure 10 is an illustration depicting a user altering a virtual object with two pairs 
of grabbing tangible tools, according to one embodiment of the present invention. 

Figure 1 1 is an illustration depicting a user erasing a region of a virtual object 
5 with an erasing tool, according to one embodiment of the present invention. 

Figure 12 is an illustration depicting a user deforming a region of a virtual object 
with a deformation tool, according to one embodiment of the present invention. 

10 Figure 13 is a flow chart showing the steps for smoothing or texturizing a virtual 

object with the smoothing tool, according to one embodiment of the present invention. 

Figure 14 is an illustration depicting the before and after of a virtual object which 
has been smoothed by a smoothing tool, according to one embodiment of the present 
15 invention. 

Figure 15 illustrates a flowchart of a player using a grabbing tool along with a 
pointing tool to create a virtual three-dimensional solid curve or volume, according to one 
embodiment of the present invention. 

20 

Figure 16 illustrates a flowchart of a player using 2 grabbing tools to rotate a 
virtual object in 3-D space, according to one embodiment of the present invention. 

Figure 17 illustrates a flowchart of a player using one or more grabbing tools and 
25 a gripping tool to place an object in 3-D space, according to one embodiment of the 
present invention. 
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Figure 18 illustrates a flowchart of a player using a first grabbing tool to grab a 
second gripping tool's virtual component, according to one embodiment of the present 
invention. 

Figure 19 illustrates a flowchart of a player using two pointing devices to locate a 
point in 3-D space, according to one embodiment of the present invention. 

Figure 20 illustrates a flowchart of a player using a grabbing tool and a gripping 
tool to cut an object located far away from the player in 3-D space according to one 
embodiment of the present invention. 
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9 



DETAILED DESCRIPTION OF THE INVENTION 



A method and an apparatus for using both physical and digital input 
methodologies to implement three-dimensional spatial manipulation and three- 
5 dimensional entertainment is described. In the following description, numerous details 
are set forth in order to provide a more thorough description of the present invention. It 
will be apparent, however, to one skilled in the art, that the present invention may be 
practiced without these specific details. In other instances, well known features have not 
been described in detail so as not to unnecessarily obscure the present invention. 

0 

Three-Dimensional Multifunction Tangible Tools 



According to one embodiment, the present invention provides for using 
multifunction tangible tools to assemble and rearrange digital objects. Physical motions 
15 of the user are mapped to interface commands through these tools. These tools are 

connected to a computer which processes the motion data (see item 305 of Figure 3, item 
605 of Figure 6, and item 705 of Figure 7). These tools allow for the motions of the body 
to become a powerful mode of interaction with virtual reality (VR). 

20 Using ordinary two-dimensional mouse interface tools to specify relative position 

and orientation of virtual objects in three-dimensional space is often difficult because the 
physical input methodology is of lower dimensionality than the task. The multifunction 
tangible tools of this embodiment solve this problem in that they allow for 6 or more 
DOF. 

25 

All of the various types of multifunction tangible tools specified in this 
embodiment can be constructed to control 6 or more DOF. There are several types of 
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controlling that these tools can be manufactured to accommodate. The types of 
controlling are as follows: 

• Controlling a physical component's 1, 2, or 3-dimensional orientation in space. 

• Controlling a physical component's position, without regard to the physical 

5 component's orientation: This controlling of position on the X, Y, and Z axis only is 
referred to as 3 DOF (See Figure 1). In Figure 1, the box represents a generic 
multifunction tangible tool. In this figure, the box's translating motions are along the X, 
Y, and Z axis. 

• Controlling a physical component's position and orientation: This controlling of 
10 position on the X, Y, and Z axis along with controlling of rotation about the X, Y, and Z 

axis is referred to as 6 DOF (See Figure 2). In Figure 2, the box again represents a 
generic multifunction tangible tool. In this figure, the box's translating motions are along 
the X, Y, and Z axis and rotating motions are around the X, Y, and Z axis, respectively. 
These rotary motions are also commonly referred to as roll, pitch, and yaw about the X, 
15 Y, and Z-axis respectively. 

According to another embodiment, the present invention is independent of the 
type of tracking technology that is used. Thus, any type of tracking technology can be 
implemented in the multifunction tangible tools of this embodiment. For example, types 

20 of tracking technology that can be used are magnetic, optical, acoustical, or inertial 

trackers. A desirable feature of this embodiment of the present invention is that a user is 
presented with tangible tools that have readily identifiable primary functions. This 
feature allows for a more spontaneous creation of three-dimensional virtual objects and a 
more intuitive gameplay because it creates a more perceptual relationship between the 

25 physical input methodology and its virtual product. These tools act like props (i.e. 

pseudo items which resemble genuine objects) because their physical components, and 
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also sometimes their virtual components, are specifically shaped to mimic their primary 
functions. 

For example, a tangible tool with its physical component shaped like a handle and 
its virtual component shaped like a sword's blade gives the user the intuitive impression 
5 that the tool is used for cutting or slicing, or a tool with its physical component shaped 
like a human hand and its virtual component shaped like a 3-dimensional paint brush 
gives the user the intuitive impression that the tool is used for painting. Thus, the 
physicality of these tangible tools allow the user to have an immediate familiarity with 
the tools' primary functions. 

10 

There is a provision of choices of physical forms that the multifunction tangible 
tools can have. Some examples of forms include, but not limited to, a physical form 
resembling a pair of tongs, a physical form resembling a gun, and a physical form 
resembling a handle (See Figures 3, 6, and 7). The physical form could also be any part 

15 of a human body. For example, a hand of a user wearing a glove can be used as a 

physical form, in which case the glove with sensors embedded in it is used to manipulate 
the hand's virtual component. Another example is a camera external to a hand shaped 
form used to manipulate the hand's virtual component. Yet another example is a device 
with infra-red capabilities external to a hand shaped form used to manipulate the hand's 

20 virtual component. 

Grabbing Device 

Item 301 in Figure 3 shows one embodiment of a physical input methodology that 
25 is shaped to resemble a pair of kitchen tongs. Grabbing device 301 comprises of foil 
sensors 302 and 303, and a potentiometer 304. The grabbing device 301 is connected to 
computer 305. This tool's primary function, which mimics the form's real-life function, 
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is to grab virtual objects. This grabbing tangible tool can be shaped to any physical form 
that resembles a set of tongs. A set of tongs is typically a grasping device consisting of 
two pieces joined at one end by a pivot or hinged like scissors. For example, the 
grabbing tool's form can resemble pincers, scissors, or tweezers. 

5 

The grabbing tangible tool's virtual form can be represented in a number of 
different ways. One way is that its virtual form can be represented by an iconic virtual 
component only, e.g., a spherical ball. Another way is that its virtual form can be 
represented by an iconic virtual component along with another virtual component that 

10 coincides with the tool's physical form, e.g., a spherical ball and a virtual pair of tongs. 
When a physical component is represented by two or more virtual components, the 
virtual components can be used to represent different things. For example, the virtual 
tongs can represent the position of the virtual component and the spherical ball can 
represent the axis of rotation of the tool. An additional way is that its virtual form can be 

15 represented by only a virtual component that coincides with the grabbing tangible tool's 
physical form, e.g., a virtual pair of tongs. Also, an alternative way is that its virtual form 
has no virtual depiction, i.e. it is neither represented by a virtual component nor a virtual 
object. 

20 In another embodiment, in addition to any of the ways described above, when the 

grabbing tangible tool's virtual component is positioned near a virtual object, an iconic 
form will react to that virtual object. This notifies the user that the virtual component is 
close enough to the virtual object to interact with it. This iconic form may be 
represented, for example by the appearance of a virtual line drawn from the virtual object 

25 to the virtual component or by the virtual object itself turning a different color. The flow 
chart in Figure 4 illustrates this process. At step 401, the position of the virtual 
component is contolled. Then, at step 402, the position of the virtual object is 
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determined. Next, at step 403, the positions of both the virtual object and virtual 
components are evaluated. Next, at step 404, a check is made to see if both virtual items 
are close enough to interact with each other. If the items are (the "yes" branch), then at 
step 405, a virtual iconic form will appear, which notifies the user that the items are close 
5 enough to interact. If, on the other hand the items are not close enough (the "no" branch), 
then continue to evaluate their positions with respect to each other at step 403. Note that 
this process can be applied to any of the tangible tools or three-dimensional tools of this 
invention. 

10 There are a number of ways of implementing controlling of the grabbing tangible 

tool's position. One way is to embed sensors inside the tongs themselves. These sensors 
can employ any available controlling technology, e.g., magnetic, optical, or inertial 
sensors. Another way is that the tongs are integrated into a controlling environment. For 
example, the tongs' physical component can be controlled by being viewed by a camera 

15 which tracks the tongs' position and/or orientation. 

According to one embodiment, sensors are placed on the inside of the tips of the 
tongs, which are located furthest from the tongs' pivot location. As an example, sensors 
302 in Figure 3 have been implemented in the tips of a pair of tongs. These sensors sense 

20 whether the tongs are closed just enough for the tong tips to touch one another. We refer 
to this closed position as the primary position. Grabbing virtual objects with the tongs in 
the primary position can be used to signify a number of different things. For example, 
the interface can be implemented to understand that the user wants to move the entire 
virtual object that is being grabbed or it can be implemented to understand that the user 

25 wants to only move a portion of the virtual object that is being grabbed. 
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Figure 3 shows another embodiment in which sensors 303 are also placed on the 
inside of the tongs closer to the tongs' pivot location. These sensors sense whether the 
tongs are closed so that the tong arms are adjacent to one another. We refer to this 
position as the secondary position. In an alternative embodiment, the controlling of how 
5 hard the tongs are being squeezed by the user can be achieved by the use of infra-red 
sensors embedded in the tong arms, pressure sensors embedded at the pivot location, a 
potentiometer 304 embedded at the pivot location, or by any other available controlling 
technology. Squeezing the tongs so that they are in the secondary position can be 
implemented to trigger certain additional functions. For example, some additional 

10 features could be a virtual menu, which exhibits a number of choices for the user to 

choose from; a shooting function, which shoots various types of virtual weapons; and a 
bomb function, which releases various types of virtual bombs. 

Figure 5 is a flow chart illustrating the steps of selecting an item from a virtual 
menu. First, at step 501, the user activates a virtual menu. After activation, at step 502, 

15 the virtual menu appears. The user can then choose an item(s) displayed by the virtual 
menu by moving the physical component towards the virtual menu item(s), which is done 
at step 503. Once the virtual item(s) is selected, that item's feature is activated at step 
504. This virtual menu item selection process can be utilized by any of the tangible tools 
or three-dimensional tools of this invention. 

20 

In addition, in another embodiment, the tongs may be manufactured to have 
additional controls which are used to activate additional features. For example, some 
types of controls could be buttons, joysticks, scroll wheels, foot pedals, or other sensors 
embedded in the device. Also, another type of control could consist of sensors embedded 
25 in the tool's stand, which is where the tool rests on when not in use. Thus, these sensors 
sense when the tool is being taken off its stand. These controls can be used to activate a 
number of various additional functions. For example, some functions could display a 
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virtual menu consisting of a number of choices for the user to choose from, toggling 
between a "default" action mode and another type of action mode, shooting virtual 
weapons of various types, and releasing virtual bombs of various types. 
Pointing Device 

5 

Item 601 in Figure 6 shows one embodiment of a physical input methodology that 
is shaped to resemble a gun. Pointing device 601 comprises of virtual beams 603 and 

604, and an additional button 602. The pointing device 601 is connected to computer 

605. This tool's primary function, which mimics its form's real-life function, is to point 
10 to virtual objects. This pointing tangible tool can be shaped to any physical form that 

resembles a real-life object that is meant to point. For example, the pointing tool's form 
can resemble, and not limited to, any gun-like form, laser pointer, camera, pointing hand, 
stick, flashlight, or spray-paint can. 

15 The specific primary function of a particular pointing tool can depend upon the 

tool's chosen physical form. For example, if the tool's physical form resembles a spray- 
paint can, the tool's specific primary function will be to spray paint of various colors onto 
three-dimensional virtual objects which the user points to with the tool. In another 
example, if the tool's physical form resembles a hot-glue gun, the tool's specific primary 

20 function will be to spray a stream of "glue" onto virtual objects, which will make the 
virtual objects appear more flexible, i.e. more rubbery, make the virtual components 
sticky, so that they stick to one another, or make the virtual objects just appear to be 
hotter in temperature, e.g., by turning the virtual objects red or orange in color. 

25 Similar to the grabbing tangible tool, the pointing tangible tool's virtual form can 

also be represented in a number of different ways. The pointing device's virtual form can 
be represented by an iconic virtual component only, an iconic virtual component along 
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with a virtual component that coincides with the tool's physical form, only a virtual 
component that coincides with the tool's physical form, or no virtual depiction at all. For 
example, the pointing device's iconic virtual component could be represented by a 
spherical ball and/or the pointing device's virtual component that coincides with the 
5 tool's physical form could be represented by either a laser beam or a stream of paint. In 
another embodiment, in addition to any of these ways, when the pointing tangible tool's 
virtual component is positioned near a virtual object, an iconic form, e.g., a virtual line 
drawn between the virtual object and the virtual component, reacts to that virtual object. 
This notifies the user that the virtual component is close enough to the virtual object to 

10 interact with it. 

Like the grabbing tangible tool, there are a number of ways of implementing 
controlling the pointing tangible tool's position. One way is to embed sensors, which can 
employ any type of available controlling technology, inside the tool itself. Another way 
is to integrate the tool into a controlling environment. The various available controlling 

15 technologies mentioned above can be used to control the tool's position and/or 
orientation in the environment. 

According to one embodiment, the pointing device's physical component is 
formed such that it has a trigger and sensors are placed inside of the trigger. These 

20 sensors can utilize any type of controlling technology, e.g., embedded magnetic, optical, 
or inertial sensors. These sensors will sense when the trigger is being pulled. Various 
types of functions can be mapped to the controlling of the trigger being pulled. For 
example, some functions could be shooting virtual objects, e.g., with virtual bullets or 
paint; pushing or pulling virtual objects, i.e. a tractor beam; slicing virtual objects using a 

25 cutting beam or laser beam that emits from the gun; and drawing connections between 
virtual objects, e.g., molecule bonds or architecture beams. Figure 6 illustrates an 
example of two feature options which draws phosphate bonds represented by thin line 
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603, and draws hydrogen bonds represented by thick line 604. Note that this drawing is a 
composite picture showing both physical objects, for example pointing tangible tool 601 
and computer 605, and virtual objects, for example laser beams 603 and 604. Also, in 
this embodiment, an additional function of property editing could be implemented to 
5 occur immediately after a particular function has occurred. For example, after the user 
has shot a virtual object, a virtual menu could be displayed to allow the user to choose 
certain properties of that particular virtual object. 

Additionally, the pointing tangible tool's physical component can be 
10 manufactured to have additional controls which can be used to activate additional 

features. For example, just as with the grabbing tangible tool, some types of controls 
could be buttons, joysticks, scroll wheels, foot pedals, or other sensors that are embedded 
into the device itself. Figure 6 shows an example of a pointing tangible tool with an 
additional action button 602. Also, another type of control could consist of sensors 
15 embedded in the tool's stand, which is where the tool rests on when not in use. These 
sensors sense when the tool is being taken off its stand. These controls can be used to 
activate a number of various additional functions. For example, some functions could be 
displaying a virtual menu consisting of a number of choices for the user to choose from, 
toggling between a "default" action mode and another type of action mode, shooting 
20 virtual weapons of various types, and releasing virtual bombs of various types. 

Gripping Device 

Item 701 in Figure 7 shows one embodiment of a physical input methodology that 
25 is shaped to resemble a handle. Gripping device 701 comprises of additional buttons 702 
and 704, and virtual beam 703. The gripping device 701 is connected to computer 705. 
This tool's primary function, which mimics its form's real-life function, is to hold, place, 
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or cut virtual objects. This gripping tangible tool can be shaped to any physical form that 
resembles a real-life object that is meant to be gripped. For example, the gripping tool's 
form can resemble any handle-like form or grip, e.g., a sword handle, laser beam handle, 
or shovel handle. 

5 

The specific primary function of a particular gripping tool can depend upon the 
tool's chosen physical form. For example, if the tool's physical form resembles a sword 
handle, the tool's specific primary function will be to cut, slice, or puncture three- 
dimensional virtual objects. A sword handle could also have a specific primary function 
10 of breaking virtual bonds of virtual DNA molecules. In another example, if the tool's 
physical form resembles a shovel handle, the tool's specific primary function will be to 
dig out three-dimensional regions of a virtual object's surface. Additionally, a gripping 
tangible tool's specific primary function could be to place virtual objects in space or to 
draw paths between virtual objects. 

15 

Similar to both the grabbing tangible tool and the pointing tangible tool, the 
gripping tangible tool's virtual form can also be represented in a number of different 
ways. In these cases, the gripping tool is used to grab a virtual component which enables 
this action. The gripping device's virtual form can be represented by an iconic virtual 

20 component only, an iconic virtual component along with a virtual component that 

coincides with the tool's physical form, only a virtual component that coincides with the 
tool's physical form, or no virtual depiction at all. For example, the gripping device's 
iconic virtual component could be represented by a spherical ball and/or the gripping 
device's virtual component which coincides with its physical form could be represented 

25 by a finite laser beam, blade, or shovel head. See Figure 7 for a depiction of an example 
of a gripping tangible tool's physical component 701 along with its finite laser beam 
virtual component 703. In another embodiment, in addition to any of these ways, when 
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the pointing tangible tool's virtual component is positioned near a virtual object, an 
iconic form, e.g., a virtual line drawn between the virtual object and the virtual 
component, will react to that virtual object. This notifies the user that the virtual 
component is close enough to the virtual object to interact with it. 

5 

Like the grabbing tangible tool and the pointing tangible tool, there are a number 
of ways of implementing controlling of the gripping tangible tool's position. One way is 
to embed sensors, which can employ any type of available controlling technology, inside 
the tool itself. Another way is to integrate the tool into a controlling environment. 
10 Various available controlling technologies can be used to control the tool's position 
and/or orientation in the environment. 

In addition, in another embodiment, the gripping tangible tool may be 
manufactured to have additional controls, which are used to activate additional features. 

15 For example, some types of controls could be buttons, joysticks, scroll wheels, foot 
pedals, or other sensors embedded in the device. Example additional buttons 702 and 
704 are depicted in Figure 7. Also, another type of control could consist of sensors 
embedded in the tool's stand, which is where the tool rests on when not in use. Thus, 
these sensors sense when the tool is being taken off its stand. Additionally, another type 

20 of control can consist of sensors, which are embedded inside the device that sense when 
the gripping tool is being held by a user. 

All of these above mentioned controls can be used to activate a number of various 
additional functions. For example, some functions could be changing the length of the 
25 virtual bladefteam, displaying a virtual menu consisting of a number of choices for the 
user to choose from, toggling between a "default" action mode and another type of action 
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mode, shooting virtual weapons of various types, and releasing virtual bombs of various 
types. 

Figure 8 shows an example of a gripping tangible tool 801 with a DNA creation 
5 virtual menu that has been activated. Figure 8 is a composite picture showing both a 
physical object, gripping device 801, and a virtual object. The virtual object comprises a 
virtual menu comprising of virtual menu icons 802 and 803, and a finite laser beam 804. 
This virtual menu presents the user with various options for creating virtual DNA 
molecules. In this example, the single-dot icon 802 of the virtual menu activates a single- 
10 strand drawing, the double-dot icon 803 of the virtual menu activates a double-helix 
drawing, and the line 803 of the virtual menu represents the finite laser beam, which is 
used to break virtual bonds. Figure 9 illustrates the user selecting the single-strand 
drawing option 802 by moving the physical component 801 towards the single-strand 
drawing virtual menu item. 

15 

Methodology For Using Two Or More Physical Input Devices In Conjunction To Alter 
Two Or Three-Dimensional Virtual Objects 

The present invention provides for using two or more physical input 
20 methodologies in conjunction to manipulate and construct two or three-dimensional 

digital objects. These physical input methodologies can consist any type of two or three- 
dimensional GUI, including any of the various types of multifunction tangible tools 
which have been specified in the previous section. 

25 Two-Handed Drawing Of Virtual Objects Using Two Or More Physical Input 
Methodologies 
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According to one embodiment of the present invention, it is possible for a user to 
use two or more physical input methodologies coincidently to draw virtual objects. 
Various types of physical input device tools may be utilized for this feature of the present 
invention. 

5 

For example, a user can use a grabbing tangible tool input device along with a 
pointing tangible tool input device to create a virtual three-dimensional solid curve or 
volume. This example employs a grabbing tangible tool formed to resemble a pair of 
kitchen tongs and a pointing tangible tool formed to resemble a gun. In this example, the 
10 user uses the tongs to grab the gun's virtual object - an emanating beam. The user then 
bends the virtual beam with the tongs. Once the beam is bent, the user sweeps the 
pointing tangible tool in space. This sweeping action causes a three-dimensional virtual 
solid curve to be created. If alternatively the user sweeps the beam so that the curve is 
closed, a three-dimensional volume is generated. 

15 

Figure 15 illustrates a flowchart of a player using a grabbing tool along with a 
pointing tool to create a virtual three-dimensional solid curve or volume. At step 1 500, a 
player uses a grabbing tool and a pointing tool to create a virtual three-dimensional curve 
or volume. At step 1501, the player grabs the pointing tool's virtual object with the 

20 grabbing tool. Next, at step 1 502, the player bends the object using the grabbing tool. 
Next, at step 1 503, a check is made to see if the virtual object is bent as needed. If the 
virtual object is not bent as needed (the "no" branch), then the player continues to bend 
the object at step 1502. If, on the other hand, the object is bent as desired (the "yes" 
branch), then at step 1504, the pointing tool is swept away from the object. Next, at step 

25 1505, another check is made to see if the sweeping action of step 1504 creates a closed 
curve. If it does (the "yes" branch), then at step 1 506, a three-dimensional volume is 
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created. If, on the other hand, the curve is not closed (the "no" branch), then at step 1507, 
a three-dimensional solid curve is created. 

According to another example, a user can use a grabbing tangible tool input 
5 device, a pointing tangible tool input device, and a gripping tangible tool input device to 
assemble and rearrange virtual molecules. In this example, the user first uses the 
gripping tangible tool's additional function of double-helix drawing to construct virtual 
DNA molecules. Once the virtual DNA molecules are constructed, the user can then use 
the pointing tangible tool to draw the DNA's phosphate bonds and hydrogen bonds. The 
10 user may use one or more grabbing tangible tools to move the virtual DNA to a position 
that allows for easier drawing of the bonds. Then, if the user desires, the user can use the 
gripping tangible tool to break any of the virtual DNA bonds. 

Two-Handed Placement Of A Virtual Object In Two Or Three-Dimensional Space 

15 

According to another embodiment of the present invention, it is possible for a user 
to use two or more physical input methodologies coincidently to change the placement of 
virtual objects in two or three-dimensional space. Various types of physical input 
methodology tools may be utilized for this feature of the present invention. 

20 

For example, a user can use two grabbing tangible tool input devices to rotate a 
virtual object in three-dimensional space. This example employs two grabbing tangible 
tools which are formed to resemble a pair of kitchen tongs. Here, the user just simply 
uses the two tools to grab the two extreme ends of the virtual object. Once the virtual 
25 object is grabbed, the user then rotates the virtual object to a desired position in space. 
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Figure 16 illustrates a flowchart of a player using 2 grabbing tools to rotate a 
virtual object in 3-D space. At step 1600, the player uses two grabbing tools to rotate a 
virtual object in 3 -dimensional space. At step 1601, the player grabs each end of object 
with the grabbing tools. Next, at step 1602, the player rotates the object to a desired 
5 shape using the grabbing tools. Next, at step 1603, a check is made to see if the object is 
rotated to the desired shape. If, it is not (the "no" branch), then the player goes back to 
step 1602 and continues to rotate the object, else (the "yes" branch) the player stops. 

In another example, the user can use one or more grabbing tangible tool input 
10 devices along with one gripping tangible tool input device to place a virtual object to a 
desired position in space. This example employs a grabbing tangible tool which is 
formed to resemble a pair of kitchen tongs and a gripping tangible tool which is formed 
to resemble a finite laser beam. In this example, the user first uses one or more grabbing 
tangible tools to rotate the virtual object to a desired position. Once the virtual object is 
15 rotated such that the virtual object's desired view is facing the user, the user can then use 
the gripping tangible tool to place the virtual object in a desired location in space. 

Figure 17 illustrates a flowchart of a player using one or more grabbing tools and 
a gripping tool to place an object in 3-D space. At step 1700, a player uses one or more 

20 grabbing tools in conjunction with a gripping tool to place an object in 3-dimensional 
space. At step 1701, the player uses the grabbing tool(s) to grab and rotate object to a 
desired position in space. Next, at step 1702, a check is made to see if the object has 
been rotated to its desired position. If not (the "no" branch), then the player goes back to 
rotating the object at step 1 701 . If, on the other hand, the object has been rotated to its 

25 desired location (the "yes" branch), then at step 1703 the player uses the gripping tool to 
move the object to the desired location in the 3-dimensional space. Next, at step 1704, 
another check is made to see if the object has been moved to its desired location. If it is 
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not (the "no" branch), then the player continues to move the object at step 1703, else (the 
"yes" branch) the player stops. 

Two-Handed Deforming Of A Virtual Object 

5 

According to another embodiment of the present invention, it is possible for a user 
to use two or more physical input methodologies coincidently to deform virtual objects. 
Various types of physical input methodology tools may be utilized for this feature of the 
present invention. 

10 

For example, a user can use two grabbing tangible tool input devices to stretch a 
virtual object. Figure 10 illustrates a user using two grabbing tangible tools, which are 
formed to resemble kitchen tongs, to stretch the sides of a virtual solid three-dimensional 
curve. Note that this drawing is a composite picture showing both physical objects, a 
15 user holding two grabbing tangible tools, and virtual objects, a virtual solid curve and two 
arrows (one towards the right of the Figure and one towards the left of the Figure) 
denoting the directions that the user is pulling the surface. 

In an additional example, a user can use two grabbing tangible tool input devices 
20 to twist a virtual object. Similar to the previously mentioned example, the user can use 
two grabbing tangible tools to twist a virtual object to a desired shape. 

Two-Handed Remapping Of Tools 

25 According to another embodiment of the present invention, it is possible for a user 

to use two or more physical input methodologies to alter one physical input device's 
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virtual component. Various types of physical input device tools may be utilized for this 
feature of the present invention. 

For example, a user can use one grabbing tangible tool input device to modify the 
5 axis of rotation of another grabbing tangible tool input device. This example employs 
two grabbing tangible tools which are both formed to resemble a pair of kitchen tongs. In 
this example, the user uses the first grabbing tangible tool to grab the second grabbing 
tangible tool's virtual component, which is shaped like a spherical ball. This virtual 
component defines the axis of rotation for the second grabbing tangible tool. Then, the 
10 user uses the first grabbing tangible tool to move the virtual component to a desired 

location in relationship to the second grabbing tangible tool. Once the virtual component 
is positioned to a desired location, the user can then use the second grabbing tangible tool 
to rotate a virtual object. 

15 Figure 1 8 illustrates a flowchart of a player using a first grabbing tool to grab a 

second gripping tool's virtual component. At step 1800, a player uses a first grabbing 
tool to grab onto a second grabbing tool's virtual component. At step 1801, the player 
uses the first grabbing tool to move the component of the second grabbing tool to a 
desired location. Next, at step 1802, a check is made to see if the component is in the 

20 desired location. If it is not (the "no" branch), then the player continues to move the 

component at step 1 801 . If, on the other hand, the object is moved to its desired location 
(the "yes" branch), then at step 1803, the player uses the second grabbing tool to rotate 
the object. 

25 Two-Handed Definition Of A Point In Three-Dimensional Space 
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According to another embodiment of the present invention, it is possible for a user 
to use two or more physical input methodologies coincidently to specify a point in two or 
three-dimensional space. Various types of physical input methodology tools may be 
utilized for this feature of the present invention. 

5 

For example, a user can use two pointing tangible tool input devices to specify a 
point in two or three dimensional space. This example employs two pointing tangible 
tools, each of which is formed to represent a gun. There are a couple of ways that these 
tools' virtual objects can be formed for this example. In one way, each pointing tangible 

10 tool has a virtual object that is formed to resemble a laser beam. In this example, the user 
physically positions the tools such that their virtual beams intersect. The intersection of 
these two virtual beams denotes a point in space. In another way, only one pointing 
tangible tool has a virtual object that is formed to resemble a laser beam and, 
alternatively, the other pointing tangible tool has a virtual object that is formed to 

15 resemble a solid plane emanating from the barrel of the gun. Similar to the prior 

example, the user physically positions the tools such that the virtual beam intersects the 
virtual plane. The intersection of the virtual beam and virtual plane signifies a point in 
space. 



20 Figure 19 illustrates a flowchart of a player using two pointing devices to locate a 

point in 3-D space. At step 1900, a player uses 2 pointing devices to locate the position 
of a point in 3-dimensional space. At step 1901, the player turns on the first pointing 
device to shoot a first laser beam in a desired direction. Next, at step 1902, the player 
turns on a second pointing device to shoot a second laser beam in a desired direction. 

25 Next, at step 1903, the player intersects first and second lasers at a desired location in 3-D 
space. Finally, at step 1904, the intersection of the two lasers signifies a point in 3-D 
space. 

0007975-0048/ CIT-3892 27 

LOSANGELES 105062v4 



Methodology For Associating Physical Components To Two or Three-Dimensional 
Virtual Components 



5 The present invention provides for altering the association between a physical 

input methodology's physical component and its corresponding two or three-dimensional 
virtual component(s). This feature allows for more flexibility in creating virtual objects. 

Displacing Virtual Component From Physical Component 

10 

According to one embodiment of the present invention, it is possible for a user to 
use one physical input methodology to alter the spatial relationship between another 
physical input methodology and its corresponding virtual component. Various types of 
physical input methodology tools may be utilized for this feature of the present invention. 

15 

Consider an example where a user wants to cut a virtual object that is located far 
from the user. To make this task easier, the user can use a grabbing tangible tool input 
device to lengthen the virtual component of a gripping tangible tool input device. In this 
example, the specific primary function of the gripping tangible tool's virtual component 

20 is to cut. This example employs a grabbing tangible tool which is shaped to resemble a 
pair of kitchen tongs and a gripping tangible tool which is shaped to resemble a finite 
laser beam handle. In this example, the user uses the grabbing tangible tool to grab the 
gripping tangible tool's virtual component, which is formed to resemble a finite laser 
beam. The user then uses the grabbing tangible tool to pull the virtual beam until it 

25 reaches a desired length. Once the desired beam length is achieved, the user can then use 
the gripping tangible tool to cut the far away virtual object. 
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Figure 20 illustrates a flowchart of a player using a grabbing tool and a gripping 
tool to cut an object located far away from the player in 3-D space. At step 2000, the 
player uses a grabbing tool and a gripping tool to cut an object in 3-D space. At step 

2001, the player uses the grabbing tool to grab the gripping tool's virtual component. 

5 Next, at step 2002, the player pulls the component until it reaches a desired length. Next, 
at step 2003, a check is made to see if the component is stretched to the desired length. If 
it is not (the "no" branch), then the player continues to stretch the component at step 

2002. If, on the other hand, the desired length of the component is reached (the "yes" 
branch), then at step 2004, the player uses the gripping tool to cut the object. 

10 

Physical Component Is Mapped To Multiple Virtual Components 



According to another embodiment of the present invention, it is possible for a user 
to map multiple virtual components to one physical component. In this feature, the user 
15 has the option to map one or more virtual components to a single physical component. 
This option can be presented to the user in a variety of ways. For example, this mapping 
option can be on a virtual menu, which can be activated by the user via an additional 
control on the physical input device. 



20 Change Which Type of Virtual Component is Mapped To The Physical Component 

According to another embodiment of the present invention, it is possible for a user 
to change the type(s) of virtual component(s) that is mapped to one physical component. 
For example, a user can change a physical input device's virtual component from a 
25 spherical ball representation to a solid cube representation. 
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In this feature, the user is able to select the type of virtual component(s) that is to 
be mapped to one physical component. This option can be presented to the user in a 
variety of ways. For example, this mapping option can be on a virtual menu, which can 
be activated by the user via an additional control on the physical input device. 

Three-Dimensional Virtual Tools 



According to one embodiment of the present invention, a number of various 
three-dimensional virtual tools can be provided to a user. These tools allow the user to 
10 create three-dimensional digital depictions of a design idea with greater ease. 



The use of commonly used two-dimensional GUIs to perform these three- 
dimensional tasks requires the designer to do much planning of the construction process 
in order to achieve a desired shape. This is because typical two-dimensional GUIs only 

15 control 2 DOF. Like the multifunction tangible tools, the three-dimensional GUIs 
presented in this embodiment can be manufactured to control 6 or more DOF. The 
controlling of 6 or more DOF, allows for more intuitive connection between a designer's 
movements of the GUI and the three-dimensional virtual product. Thus, in contrast to 
typical two-dimensional GUIs, these three-dimensional GUIs are better able to capture 

20 the designer's gesture, emotion, and spontaneity. 



The three-dimensional virtual tools of this embodiment can be shaped to any 
physical form. For example, these tools may be formed to resemble specific 
commonplace items or may be formed into amorphous ergonomic shapes that fit 
25 comfortably in the user's hand. 
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Similar to the multifunction tangible tools, the virtual form of any of the three- 
dimensional virtual tools in this embodiment can be represented in a number of different 
ways. The virtual form can be represented by an iconic virtual component only, an iconic 
virtual component along with a virtual component that resembles the tool's physical 
5 form, only a virtual component that resembles the tool's physical form, or no virtual 
depiction at all. For example, an iconic virtual component could be represented by a 
spherical ball. In another embodiment, in addition to any of these ways, when the virtual 
component of any of these tools is positioned near a virtual object, an iconic form, e.g., a 
virtual line drawn between the virtual object and the virtual component, will react to that 
10 virtual object. This notifies the user that the virtual component is close enough to the 
virtual object to interact with it. 

There are a number of ways of implementing controlling the position of the three- 
dimensional virtual tools of this embodiment. One way is to embed sensors, which can 
15 employ any type of available controlling technology, inside the tool itself. Another way is 
to integrate the tool into a controlling environment. Various available controlling 
technologies can be used for this way to control the tool's position and/or orientation in 
the environment. For example, types of tracking technology that can be used, and not 
limited to, are magnetic, optical, acoustical, or inertial trackers. 

20 

According to another embodiment, other types of controlling can be employed in 
the three-dimensional virtual tools. One type of controlling could consist of sensors 
embedded in the tool's stand, which is where the tool rests on when not in use. Thus, 
these sensors sense when the tool is being taken off its stand. Another type of controlling 
25 can consist of sensors, which are embedded inside of the device that sense when the tool 
is being held by a user. All of the above mentioned different types of controlling can be 
used to activate a number of various functions. 
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Eraser Tool 



According to one embodiment, the three-dimensional virtual tool of the present 
invention is an eraser tool. This tool's primary function, similar to its real-life function, 
5 is to erase three-dimensional regions of virtual objects. Once a virtual drawing has been 
made, a user can remove portions of the drawing by waving the eraser tool on the areas of 
the virtual object that the user wants to erase. The size of the three-dimensional region 
that the eraser removes can be set to a default predetermined size. Figure 1 1 shows a user 
using an eraser tool 1 101 to erase a three-dimensional region (the region is depicted as a 
10 white region over a gray three-dimensional region) of a virtual object 1 102 that is shaped 
to resemble a chair. Note that this illustration is a composite picture showing both 
physical objects, the user holding an eraser tool 1101, and a virtual object, a virtual chair 
1102. 

15 According to one embodiment, the eraser tool's physical component can be 

manufactured to have additional controls which can be used to activate additional 
features. For example some types of controls could be, and not limited to, buttons, 
joysticks, scroll wheels, foot pedals, or other sensors that are embedded into the 
methodology itself. The other sensors that are embedded in the methodology itself could 

20 be used to control, for example, when the user applies pressure to a device with a thumb. 
These additional controls can be used to activate a number of various additional 
functions. For example, some additional functions could be changing the size of the 
default erasing region or displaying a virtual menu consisting of a number of choices 
from which the user can choose. Another function could be toggling between a "default" 

25 erasing mode and another type of erasing mode. In this example, the "default" erasing 
mode could set the eraser to erase a default region size and the other type of erasing mode 
could set the eraser to erase a user-specified region size. 
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Deformation Tool 



According to another embodiment, the three-dimensional virtual tool of the 
5 present invention is a deformation tool. This tool's primary function is to deform regions 
of three-dimensional virtual objects. This tool allows a user to deform portions of a 
virtual object by waving the deformation tool close to or on areas of the virtual object that 
the user wants to deform. If the user waives the tool near the virtual object without 
touching the virtual object, the area of the virtual object that the deformation tool is near 

10 will pull upward towards the tool. Conversely, if the user waives the tool on an area of 
the virtual object, the area of the virtual object that the deformation tool touches will push 
away from the tool. The amount of deformation of the virtual object is dependent upon 
how far away from the virtual object the user positions the tool or how deeply into the 
virtual object the user waives the tool. Figure 12 shows a user using a deformation tool 

15 to deform a three-dimensional virtual object 1201 . In this figure, the user is positioning 
the deformation tool such that the virtual object 1201 is being depressed by the tool. 
Note that this illustration is a composite picture showing both physical objects, the user 
holding a deformation tool, and a virtual object 1201 . 

20 According to another embodiment, similar to the eraser tool, the deformation 

tool's physical component can be manufactured to have additional controls which can be 
used to activate additional features. For example some types of controls could be 
buttons, joysticks, scroll wheels, foot pedals, or other sensors that are embedded into the 
methodology itself. The other sensors that are embedded in the methodology itself could 

25 be used, for example, to sense when the user applies pressure to a device with his thumb. 
These additional controls can be used to activate a number of various additional 
functions. For example, some functions could be changing the sensitivity of deformation, 
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i.e. how much deformation per unit of tool movement; displaying a virtual menu 
consisting of a number of choices from which the user can choose; or toggling between a 
"default" action mode and another type of action mode. 

5 Smoothing Tool 

According to another embodiment, the three-dimensional virtual tool of the 
present invention is a smoothing tool. This tool's primary function is to smooth or 
texturize regions of three-dimensional virtual objects. This tool allows a user to smooth 

10 or texturize portions of a virtual object by waving the smoothing tool on areas of the 
virtual object that the user wants to smooth or texturize. The size of the three- 
dimensional region that the smoother tool smoothes or texturizes can be set to a default 
predetermined size. The tool uses an algorithm to uniformly smooth or texturize a 
selected area of a virtual object. Figure 13 illustrates the smoothing or texturizing 

15 process that the smoothing tool undergoes. First, at step 1301, the user uses the physical 
component to touch an area of a virtual object that is to be smoothed or texturized. Once, 
a region of a virtual object is touched with the physical component, then at step 1302 an 
algorithm uniformly smoothes or texturizes the touched portion of the virtual object. 
Then, at step 1303, the region of the virtual object will appear to be smoothed or 

20 texturized. Figure 14 shows a digital illustration of two three-dimensional virtual 

drawings of a human head. Head 1401 depicts a virtual head before being smoothed by 
the smoothing tool, and head 1402 depicts a virtual head after being smoothed by the 
smoothing tool. 

25 According to one embodiment, the smoother tool's physical component can be 

manufactured to have additional controls which can be used to activate additional 
features. For example some types of controls could be, and not limited to, buttons, 
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joysticks, scroll wheels, foot pedals, or other sensors that are embedded into the 
methodology itself. The other sensors that are embedded in the methodology itself could 
be used to control, for example, when the user applies pressure to a device with a thumb. 
These additional controls can be used to activate a number of various additional 
5 functions. For example, some functions could be changing the size of the default 

smoothing region, changing the degree of smoothing or texturizing the tool will perform, 
or displaying a virtual menu consisting a number of choices from which the user can 
choose. Another function could be toggling between a "default" action mode and another 
type of action mode. In this example, the "default" action mode could set the smoother to 
10 smooth mode and the other type of action mode could set the smoother to texturize mode. 
Alternatively, in this example, the "default" action mode could set the smoother to 
smooth or texturize a default region size and the other type of action mode could set the 
smoother to smoother or texturize a user-specified region size. 

15 

Painting Tool 

According to another embodiment, the three-dimensional virtual tool of the 
present invention is a spray-painting tool. This tool's primary function is to spray various 

20 colors of virtual paint onto regions of three-dimensional virtual objects. This tool allows 
a user to spray paint a portion of a virtual object by pointing the spray-painting tool to the 
area of the virtual object that the user wants to paint. The size of the three-dimensional 
region of the virtual object that the spray-painting tool covers with paint is dependent 
upon the distance the user holds the tool from the virtual object. The rate of flow of paint 

25 being sprayed by the spray-painting tool is set to a default predetermined setting. 
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In one embodiment, the spray-painting tool's physical component can be 
manufactured to have additional controls which can be used to activate additional 
features. For example some types of controls could be buttons, joysticks, scroll wheels, 
foot pedals, or other sensors that are embedded into the methodology itself. The other 
5 sensors that are embedded in the methodology itself could be used to sense, for example, 
when the user applies pressure to a device with a thumb. These additional controls can 
be used to activate a number of various additional functions. For example, some 
functions could be changing the color of the paint being sprayed, changing the flow rate 
of the paint being sprayed, or displaying a virtual menu consisting a number of choices 

10 from which the user can choose. Another function could be toggling between a "default" 
action mode and another type of action mode. In this example, the "default" action mode 
could set the spray-painting tool to spray a default color of paint and the other type of 
action mode could set the spray-painting tool to spray a different color of paint. 
Alternatively, in this example, the "default" action mode could set the spray-painting tool 

15 to spray paint at a default flow rate and the other type of action mode could set the spray- 
painting tool to spray paint at a user specified flow rate. 

Texture Tool 

20 According to another embodiment, the three-dimensional virtual tool of the 

present invention is a texture-spraying tool. This tool's primary function is to spray 
various amounts of virtual texture onto regions of three-dimensional virtual objects. This 
tool allows a user to spray texture on a portion of a three-dimensional virtual object by 
pointing the texture-spraying tool to the area of the virtual object that the user wants to 

25 texturize. The size of the three-dimensional region of the virtual object that the texture- 
spraying tool texturizes is dependent upon the distance the user holds the tool from the 
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virtual object. The rate of texture that being sprayed by the texture-spraying tool is set to 
a default predetermined setting. 

According to one embodiment, the texture-spraying tool's physical component 
5 can be manufactured to have additional controls which can be used to activate additional 
features. For example some types of controls could be, and not limited to, buttons, 
joysticks, scroll wheels, foot pedals, or other sensors that are embedded into the 
methodology itself. The other sensors that are embedded in the methodology itself could 
be used to sense, for example, when the user applies pressure to a device with a thumb. 

10 These additional controls can be used to activate a number of various additional 

functions. For example, some functions could be changing the type of texture being 
sprayed, changing the amount of texture being sprayed, or displaying a virtual menu 
consisting a number of choices from which the user can choose. Another function could 
be toggling between a "default" action mode and another type of action mode. In this 

15 example, the "default" action mode could set the texture-spraying tool to spray a default 
type of texture and the other type of action mode could set the texture-spraying tool to 
spray a different type of texture. Alternatively, in this example, the "default" action mode 
could set the texture-spraying tool to spray texture at a default rate and the other type of 
action mode could set the texture-spraying tool to spray texture at a user specified rate. 

20 

Three-Dimensional Video Game Controllers 

According to one embodiment, the present invention provides for a number of 
three-dimensional video game controllers. These controllers allow a new class of video 
25 games in which the player directly manipulates three-dimensional objects. 
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Video games have recently found the ability to render three-dimensional form 
using advanced computer hardware, as seen in many games released since the mid- 
1990 's. In addition, there are many new games that use devices such as dance pads and 
snowboard as controllers to enhance video games. The present invention allows for 
5 commonplace methodologies to more richly interact with the three-dimensional graphics. 
These methodologies are applications of the tools developed for spatial design in the area 
of video gaming. 

Grabbing Controller - Tongs In A Game Scenario 

10 

The tongs closing around a shape can signal many events in a game. For 
example, a game has enemies fly towards a player. Closing the tongs around the enemies 
can, for example, kill them and gain the user points or help the user advance to the next 
level. 

15 

The grabbing controller can be used for other forms of manipulation that entail 
rotating and moving objects as well. For example, another game involves three- 
dimensional puzzle pieces. In this game the player grabs the puzzle pieces with the tongs 
and links them together to form larger structures. In some cases these pieces move, in 

20 other cases the pieces do not. In other variants of this game, the player can also bend and 
twist these objects, both by grabbing portions of an articulated skeleton with one tong, or 
by grabbing objects with two tongs. When grabbed with two tongs, objects are 
deformed, bent, scaled, and stretched for the purposes of the game, for example. For a 
game which uses this interaction, a player gets points for bending a shape to match a 

25 target form, and as this game advances, the player has to bend more complex shapes 
more quickly to gain points. 
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Other games that demonstrate the usefulness of this grabbing controller include a 
role playing game where many objects can be grabbed during the course of the game. In 
one example, the player navigates a three-dimensional world, grabbing objects such as 
weapons, food, gold, and clothing to enhance the abilities of a character in the game. In 
5 another game, this grabbing technique is used to arrange things to solve puzzles. In 
another game, players have to place objects in certain locations in a room to unlock a 
door and allow them to go into the next room. In yet another puzzle game, the objects 
have to touch one another, for example, to form complex objects. 

10 Slicing And Placing With A Handle In A Game Scenario 

The handle (physical component) with a laser beam (virtual component) can be 
used in a variety of games. For example, the device can be used in a game as a sword to 
fight an opponent. In this game, the opponent is a rendered three-dimensional digital 

15 character, and the opponent is damaged when the virtual component (laser beam) goes 
through the character. In another type of game, this sword is used to kill much smaller 
enemies. In another game, these characters appear as a swarm of flies, and the player is 
rewarded for moving the laser beam through a fly. The objective of this game is to kill 
insects, which could come in differing varieties and complexities as the player advances 

20 through the different levels of the game. In this type of game the remapping technique 
can be used, for example, to change from one type of sword to another or to cycle 
through an array of a variety of weapons. There is virtually no limitation to the range of 
weapons that such a handle can host. 

25 The handle can also be used to drop objects in a scene as part of playing the game. 

For example, the objective of a game could be to destroy buildings using a variety of 
bombs. In this game, a player has to drop explosives on a building and destroy them in 
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order to advance in the game. The player moves through the world using a navigation 
device similar to any number of navigational devices described earlier (grabbing, 
pointing, or gripping devices) to places bombs in certain areas. Then a tool such as the 
handle is used to place the bombs, and the menu described earlier is used to select the 
5 type of explosive that will be dropped. There is no limit to the nature of objects that can 
be placed with such a tool. While objects are commonly placed in video games, this tool 
allows them to be placed more richly by specifying all 6 DOFs as they are placed. This is 
one of the advantages of this style of interface for video games. 

10 Shooting And Selecting With A Pointing Device In A Game Scenario 

The pointing device described above can be used to shoot objects in a video 
game. For example, the objective of a game could be to shoot down the enemy. In this 
game, bullets come flying out of a gun. Unlike previous shooting video games where the 

15 user points at a screen, the present invention does not specify a two-dimensional screen 
coordinate for the shot, but rather a ray in three-dimensional space, which is a crucial 
distinction between the present invention and the prior art. In another game, the user uses 
the pointing device to select between a range of weapons to use against enemies. These 
weapons include, for example, a wide variety of real and fictional guns, flamethrowers, 

20 laser shooters, and weapons that hurl any variety of objects as projectile weapons. 

Furthermore the pointing device can be used to select objects. For example, a 
tractor beam virtual component selects an object and as the player holds the trigger, the 
object moves towards the player. This can be used to grab or rearrange objects. Other 
25 games can be built on the use of a gun to highlight certain things, or to choose them. For 
example, a city-simulation strategy game can use this device to select a building. Once it 
is selected, a user can change the properties of the building, such as deciding to convert it 
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into a factory, or to raise the rent of the apartments in this building. In this manner the 
pointing device can be used to point to three-dimensional game components during the 
course of playing the game. The pointing device can also be used to hold a beam that it 
moves to slice objects, or paint objects, which can be used in the two-handed methods 
5 described above. 

DrawinR Strokes In A Game Scenario 



Drawing strokes, either with the hand, or through a much simpler method using 
10 the handle (where a one-dimensional curve is traced in three-dimensional space) can be 
used as an element of gameplay for three-dimensional immersive entertainment. For 
example, in one game the player draws strokes to match target strokes. The target strokes 
appear in a series, one after the other. After each target stroke appears, the player has 
five seconds to draw this target shape. If the player is successful in drawing the target 
15 shape, the game proceeds to the next level where the shapes become more complex. 

Strokes can be used in a wide variety of three-dimensional video games, for 
example in games where the objective is to draw a stroke around an enemy form. For 
example, the player draws strokes to block the path of an enemy robot. Each stroke 
serves as a boundary which forces the robot to turn around. The objective here is to draw 
20 a series of strokes to trap the robot. This game shows how strokes can be used as objects 
which other characters interact with. This is useful for games which entail creating part 
of a world as a portion of gameplay. 



Navigation In A Game Scenario 

25 

The present invention provides a rich means for navigating a player through a 
video game world, and many other manipulations of game space. For example, the tongs 
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described earlier can be used to grab a world inside a game and move it, or when a player 
grabs empty space and moves the tongs, the world moves as well. The player, seeing the 
world in a first person perspective, perceives to be moving through the world. The device 
can also be used to guide a vehicle that the player perceives to be in. For example, the 
5 handle controls the navigation of a plane. Moving the handle to the left makes the plane 
bank to the left, and the player views a changing scenery to reflect the change in direction 
similar to what the player would view in a real-life situation. Moving the handle forward 
or backwards can control the speed of the plane. Note this is also effective in situations 
when the player is not controlling a plane, but another device. This is also effective 
10 when, instead of the handle, a hand, or a form of a plane (for example, a small toy plane 
with sensors in it) is used as an input device. A similar type of navigation can be used to 
move a player, for example, through tunnel-like spaces, or along a racetrack. In this type 
of game a device is waved to signify the direction and velocity of the player along the 
path, and the character in the video game responds accordingly. 

15 

Non-Controller-Specific Three-Dimensional Interactions In A Game Scenario 

The present invention embodies other interactions that do not depend on a certain 
form of controller, but are still unique to the three-dimensional gaming setting. For 

20 example, in an interactive video game where an enemy flies towards the player's input 
device, and the player must move the form away from the enemy. There are many such 
styles of play which are supported by the present invention. In another example video 
game scenario, a weapon lies on a rubber band that is between two devices. Moving the 
input device moves the rubber band and the attached weapon. Similarly weapons can be 

25 dangled from virtual components that look like strings and swung at players. 
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Another class of interactions with the gaming world involves specifying regions 
of space within a game domain. In one such game, the player draws toxic areas in three- 
dimensional space. In this game, the user (who is representing the n good guy" in the 
game) is bombarded by flying enemies. One of the tools the user has allows the user to 
5 draw a region such that if the enemy flies through this region, the enemy is killed. This is 
one of many ways a user can make use of the improved interface tools of the present 
invention Another way of playing this game is for the user to draw three-dimensional 
windows in space analogous to the region above. The player activates the input device, 
drags it, and releases it. A square is formed spanning the start and end points, and this 
10 square forms a window. If the enemy goes through this window they disappear. 

Thus, a method and an apparatus for using both physical and digital input 
methodologies to implement two and three-dimensional spatial manipulation is described 
in conjunction with one or more specific embodiments. The invention is defined by the 
15 following claims and their full scope of equivalents. 
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